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ABSTRACT 


In  this  thesis,  the  feasibility  of  developing  a  pen  input  device  using  an 
inertial/magnetic  sensor  module  is  investigated.  The  emphasis  is  on  testing  and 
evaluation  of  algorithms  for  computing  handwriting  trajectories  based  on  accelerometer 
measurement  data.  This  research  starts  by  placing  the  inertial/magnetic  sensor  in  a  2-D 
plane  and  writing  alphanumeric  characters.  Before  continuing  to  evaluate  the  3-D 
writing,  a  calibration  algorithm  is  implemented  for  computing  the  length  between  the 
nose  of  the  pen  input  device  and  the  point  where  the  inertial/magnetic  sensor  is  attached 
to  the  pen.  A  velocity  correction  algorithm  is  also  applied  by  recognizing  the  pause 
phases  in  the  writing  in  order  to  eliminate  the  drift  in  acceleration  measurements  and 
accurately  reproduce  handwriting  trajectories.  Extensive  experiments  conducted  for  2-D 
and  3-D  writings  indicate  that  it  is  possible  to  develop  a  pen  input  device  to  track 
handwriting  using  an  inertial/magnetic  sensor  module.  However,  the  performance  of  the 
handwriting  tracking  depends  on  the  accuracy  of  the  sensor  module  and  the  speed  of  the 
writing  motion. 
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EXECUTIVE  SUMMARY 


The  development  of  Micro-Electro-Mechanical  Systems  (MEMS)  technology  has 
created  interest  in  developing  many  kinds  of  input  devices  for  desktop  computers  and 
laptop  computers.  This  thesis  research  investigates  the  feasibility  of  developing  a  pen 
input  device  that  can  reproduce  the  handwriting  trajectory  of  alphanumeric  characters. 
This  pen  input  device  is  small,  efficient,  useful  and  comfortable  for  writing.  It  is 
constructed  by  attaching  an  inertial/magnetic  sensor  module  to  a  pen. 

The  inertial/magnetic  sensor  module  used  in  this  study  is  a  MicroStrain  3DM- 
GX1.  The  sensor  module  contains  three  orthogonal  accelerometers,  three  orthogonal 
magnetometers,  and  three  orthogonal  gyrometers  for  each  axis  x,y,z  and  operates  over 
360  degrees.  It  provides  acceleration,  angular  rate,  and  magnetometer  measurements 
while  the  pen  tracks  the  alphanumeric  characters. 

The  acceleration  measurements  with  double  integration  provide  the  position  of  the 
pen  and  consequently  the  handwriting  trajectory.  The  magnetometer  data  and  the 
acceleration  data  using  the  factored  quaternion  algorithm  (FQA)  provide  the  orientation 
quaternion  of  the  sensor  module  and  consequently  of  the  pen  input  device.  For  the 
handwriting  tracking  of  alphanumeric  characters  in  2-D  plane  and  3-D  space  as  presented 
in  this  thesis,  the  sensor  quaternion  was  used  instead  of  the  quaternion  provided  from  the 
FQA. 

The  estimated  position  of  the  pen  input  device  must  be  computed  in  the  earth 
coordinate  system.  However,  the  sensor  module  provides  acceleration  measurements  in 
the  sensor  coordinate  system.  Consequently,  the  acceleration  data  must  be  converted  to 
the  earth  coordinate  system  and  this  conversion  is  implemented  by  using  the  quaternion. 

In  this  thesis,  an  experiment  demonstrating  the  importance  of  the  sensor  module 
velocity  for  the  reproduction  of  an  accurate  handwriting  trajectory  was  conducted.  The 
sensor  module  was  placed  in  a  2-D  plane  and  moved  to  track  a  straight  line  with  three 


xv 


different  motion  velocities.  The  numerical  and  graphical  results  from  the  experiment 
indicate  that  the  accurate  handwriting  trajectory  is  accomplished  only  by  using  the  sensor 
module  with  high  motion  velocity. 

In  this  research,  a  velocity  correction  algorithm  is  also  described.  An  experiment 
with  the  sensor  module  took  place  and  indicated  with  graphs  that  prior  to  the  sensor 
module  motion  and  after  the  sensor  module  motion,  the  sensor  module  velocity  is 
nonzero.  This  is  an  error  and  the  integration  of  this  error  results  in  an  inaccurate 
handwriting  trajectory.  Mathematical  equations  were  applied  for  the  velocity  correction 
algorithm  and  after  the  implementation  of  them  in  Matlab,  an  accurate  handwriting 
trajectory  was  accomplished. 

A  calibration  algorithm  is  also  explained  in  this  study.  The  calibration  algorithm 
is  necessary  for  the  computation  of  the  velocity  in  the  nose  of  the  pen  and  eventually  for 
the  estimation  of  the  position  in  the  nose  of  the  pen  in  the  earth  coordinate  system.  This 
algorithm  provides  the  constant  length  vector  between  the  nose  of  the  pen  input  device 
and  the  point  where  the  inertial  magnetic  sensor  is  attached  to  the  pen.  The  quaternion 
provided  from  the  factored  quaternion  algorithm  (FQA)  was  used  for  the  calibration 
algorithm.  In  this  thesis  research,  the  sensor  module  was  attached  to  a  pen  at  a  distance  of 
14  cm  from  the  nose  of  the  pen.  After  the  implementation  of  the  equations  that  describe 
the  calibration  process  in  Matlab,  the  constant  length  was  computed  to  be  14. 1  cm. 

Real  experimental  handwriting  results  of  alphanumeric  characters  for  handwriting 
tracking  in  2-D  plane  and  3-D  space  are  presented  in  this  thesis  research.  It  is  observed 
from  all  the  handwriting  results  that  the  reproduced  handwriting  trajectory  is  qualitatively 
correct  and  recognizable  by  humans.  This  indicates  that  the  construction  of  this  pen  input 
device  using  an  inertial/magnetic  sensor  for  the  reproduction  of  the  handwriting 
trajectory  of  alphanumeric  characters  is  feasible. 

It  is  obvious  that  errors  in  the  orientation  quaternion  will  affect  the  accuracy  of 
the  acceleration  data  in  the  earth  coordinate  system.  The  sensor  specifications  state  that 
the  dynamic  accuracy  of  the  sensor  is  +1-2  degrees  for  arbitrary  orientation  angles. 
Consequently,  the  sensor  does  not  provide  the  accurate  orientation  quaternion  and  thus  of 
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the  pen  while  the  handwriting  tracking  is  taking  place.  Therefore,  the  inaccurate 
quaternion  creates  errors  in  the  acceleration  data  and  accordingly  after  the  double 
integration  to  the  handwriting  trajectory. 

In  this  thesis,  the  handwriting  trajectory  was  accomplished  by  using  the  sensor 
module  with  high  motion  velocity.  However,  the  performance  of  tracking  handwritten 
alphanumeric  characters  in  3-D  space  with  a  high  sensor  motion  velocity  is  really 
difficult,  especially  for  complicated  alphanumeric  characters.  Therefore,  the  handwriting 
tracking  is  not  accurate  with  the  typical  writing  speed. 

Although  this  thesis  research  shows  that  the  development  of  a  pen  input  device  by 
using  an  inertial/magnetic  sensor  is  feasible,  the  use  of  a  different  inertial/magnetic 
sensor  that  will  be  able  to  provide  more  accurate  orientation  quaternion  and  acceleration 
data  is  recommended  for  the  accomplishment  of  accurate  handwriting  trajectories. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Due  to  the  recent  development  in  Micro-Electro-Mechanical  Systems  (MEMS) 
technology,  there  has  been  an  interest  in  developing  new  input  devices  for  computers. 
There  are  various  kinds  of  input  devices  available  for  desktop  computers  and  laptop 
computers,  including  wireless  optical  mice  that  are  very  useful  in  everyday  life.  Apart 
from  that,  there  is  another  input  device  for  computers,  which  is  called  a  pen-type  input 
device. 

A  pen- type  input  device  can  reproduce  the  3-D  handwriting  trajectory  and 
reconstruct  this  handwriting  trajectory  to  a  2-D  plane.  This  input  device  can  be  very 
useful  for  professors  who  are  teaching  on-line  courses.  Professors  using  this  kind  of  input 
device  for  writing  words,  sentences,  and  many  alphanumeric  characters  will  be  able  to 
reproduce  this  handwriting  trajectory  to  desktop  computers,  laptop  computers  and 
generally  to  students’  computers.  For  distance  learning  in  the  future,  it  will  not  be 
necessary  for  professors  to  use  cameras  or  other  recording  programs,  which  sometimes 
can  not  show  the  image  of  the  lecture  clearly  because  of  the  internet  connection  or  many 
times  the  words,  sentences,  and  the  alphanumeric  characters  are  hidden  while  the 
professor  is  writing  on  the  blackboard. 

There  are  two  types  of  pen  input  devices.  There  is  the  self-contained  type  and  the 
external  reference  type.  The  external  reference  type  requires  an  external  source  such  as  a 
particular  surface  that  can  track  the  position  of  the  pen  while  the  pen  creates  the  writing 
trajectory.  The  self-contained  type  does  not  require  any  external  sources,  so  it  is  more 
useful,  comfortable,  and  superior  than  the  external  reference  type. 

There  is  an  external  reference  type  that  uses  infrared  (IR)  technology  in  order  to 
reproduce  the  handwriting  trajectory  [1],  [2].  It  is  very  useful  and  successful,  but  its 
functionality  has  a  restriction  in  its  range  of  movement.  That  means  that  this  pen  input 
device  can  be  used  only  in  limited  spaces  and  not  in  spaces  such  as  blackboards. 
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Consequently,  this  type  of  pen  input  device  can  not  be  useful  and  successful  for 
reproducing  the  handwriting  trajectory  in  on-line  courses  or  in  any  other  applications  in 
everyday  life. 

To  the  contrary,  this  thesis  research  presents  a  smart  pen  input  device,  which  can 
also  reproduce  the  handwriting  trajectory.  It  is  self-contained  because  it  does  not  require 
any  external  devices.  It  is  called  smart  because  it  is  small,  efficient,  useful,  comfortable 
for  writing,  and  can  also  reproduce  the  handwriting  trajectory  everywhere  without  having 
any  function  restrictions  such  as  limited  spaces.  This  pen  is  equipped  with  a  sensor, 
which  is  shown  in  Figure  1. 


Figure  1.  Smart  Pen  Input  Device. 


The  sensor  that  is  used  in  this  thesis  research  is  called  the  MicroStrain  3DM-GX1 
and  it  is  shown  in  Figure  2,  and  its  specification  is  provided  in  Appendix. 


Figure  2.  MicroStrain  3DM-GX1  Sensor  [From  3]. 
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It  is  composed  of  three  orthogonal  accelerometers,  three  orthogonal  magnetometers,  and 
three  orthogonal  gyros  for  the  three  axes  x,y,z  and  it  operates  over  360  degrees.  The 
sensor  collects  acceleration,  angular  rate,  and  magnetometer  data  for  the  three  axes  x,y,z, 
while  the  pen  tracks  the  writing  trajectory.  These  data  are  provided  by  the 
accelerometers,  gyros,  and  magnetometers  of  the  sensor  from  the  user’s  handwriting 
motion  in  3-D  space. 

The  acceleration  data  with  double  integration  yield  the  position  of  the  pen  and 
consequently  represent  the  handwriting  trajectory  along  with  the  time.  The  magnetometer 
data  with  the  acceleration  data  are  used  in  the  factored  quaternion  algorithm  (FQA)  as 
described  in  [4],  in  order  to  estimate  the  orientation  quaternion. 

The  factored  quaternion  algorithm  produces  the  quaternion  as  an  output.  The 
quaternion  is  used  to  represent  the  orientation  of  objects  in  3-D  space.  The  factored 
quaternion  algorithm  provides  the  orientation  of  the  sensor  and  consequently  the  pen 
input  device.  In  this  thesis  research,  the  sensor  quaternion  was  used  for  the  reproduction 
of  the  handwriting  results  in  2-D  and  3-D  plane  as  described  in  Chapters  V  and  VI.  The 
quaternion  provided  from  the  factored  quaternion  algorithm  was  used  only  in  the 
calibration  algorithm  as  described  in  Chapter  III. 

This  sensor  also  provides  angular  velocities  of  the  three  axes  by  the  three  gyros, 
because  the  handwriting  motion  is  not  only  a  straight  motion  but  also  a  rotation  motion. 
The  following  chapters  describe  how  the  angular  velocity  to  is  required  in  the 
implementation  of  many  mathematical  equations  in  order  to  represent  the  handwriting 
trajectory. 

From  the  many  mathematical  equations  that  will  be  described  in  the  following 
chapters  in  order  to  explain  the  main  algorithm  for  the  representation  of  the  handwriting 
trajectory,  it  is  obvious  that  the  acceleration  and  the  quaternion  or  the  orientation  of  the 
pen  are  crucial  for  the  accurate  position  of  the  pen,  and  consequently  the  reproduction  of 
the  writing  trajectory. 
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In  this  thesis  research,  some  experiments  that  took  place  proved  that  the  sensor 
not  only  does  not  provide  the  accurate  acceleration  and  consequently  the  position  of  the 
pen,  but  also  that  the  velocity  of  the  sensor  motion  is  crucial  for  the  accurate  acceleration 
and  position  of  the  pen.  A  more  detailed  description  for  this  conclusion  is  provided  in 
Chapter  II. 

Apart  from  that,  the  position  of  the  pen  must  be  calculated  in  the  earth  coordinate 
system.  By  convention  the  earth  coordinate  system  has  the  x-axis  pointing  to  north,  the  y- 
axis  pointing  to  east  in  the  horizontal  plane  and  the  z-axis  “straight  down  toward  the 
center  of  the  earth.”  [5].  Also,  we  define  the  sensor  coordinate  system  with  the  x-axis,  y- 
axis  and  z-axis  in  reference  to  the  sensor.  The  earth  coordinate  system  and  the  sensor 
coordinate  system  are  shown  in  Figure  3. 


Figure  3.  Sensor  Coordinates  xs  -  ys  -  zs  and  Earth 

Coordinates  xe  -  ye  -  ze . 

However,  the  acceleration  data  provided  from  the  sensor  are  in  the  sensor 
coordinate  system  and  consequently  must  be  converted  to  the  earth  coordinate  system.  It 
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will  be  described  in  the  main  algorithm  for  the  reproduction  of  the  writing  trajectory  that 
this  conversion  can  be  implemented  by  using  the  quaternion.  Consequently,  it  is  obvious 
that  if  there  are  errors  in  the  quaternion  then  the  converted  acceleration  data  in  the  earth 
coordinate  system  is  not  correct,  which  means  that  the  position  of  the  pen  and  the  writing 
trajectory  is  not  accurate. 

The  factored  quaternion  algorithm  (FQA)  as  described  in  [4]  does  not  use  any 
angular  rate  measurements,  and  consequently  it  provides  the  accurate  quaternion  for  only 
the  static  or  slow  motions  of  objects.  It  is  obvious  that  the  hand’s  motion  and  thus  the 
sensor  motion  is  not  static  and  sometimes  not  slow  when  a  person  is  writing  letters, 
words,  and  sentences  using  a  pen.  Hence,  the  quaternion  provided  from  the  FQA  is  not 
sufficient  for  real-time  hand  motion  tracking. 

Besides  the  accuracy  of  the  acceleration  and  the  quaternion,  there  is  another  topic, 
called  velocity  correction  that  affects  the  accuracy  of  the  position  of  the  pen.  The  velocity 
data  provided  by  the  integration  from  the  acceleration  data  of  the  sensor  has  an  error 
which  grows  along  with  time.  This  error  is  called  drift  and  happens  for  two  main  reasons. 
The  first  reason  happens  because  of  the  integration  of  data  prior  to  the  sensor  motion  and 
after  the  sensor  motion,  or  in  the  interval  between  the  letters  in  writing  a  word.  The  other 
reason  occurs  because  of  the  noise  of  the  sensor.  The  following  chapter  explains  that  the 
noise  of  the  sensor  is  very  crucial  and  affects  the  accurate  reproduction  of  the 
handwriting  trajectory.  Consequently,  for  these  two  main  reasons,  the  velocity  data  is  not 
correct  and  needs  correction  for  the  accurate  position  of  the  pen. 

In  this  thesis  research,  a  more  detailed  description  with  results  for  the  topics  such 
as  quaternion,  acceleration,  and  velocity  correction  that  affect  the  accuracy  of 
handwriting  trajectory  is  provided  in  the  following  chapters. 

Experimental  handwriting  results  in  2-D  plane  and  3-D  space  are  also  included  in 
this  thesis.  The  handwriting  trajectory  in  2-D  plane  is  reproduced  when  the  sensor  is 
placed  on  a  desk  as  shown  in  Figure  4  and  the  perfonnance  of  handwriting  tracking  in  3- 
D  for  the  reproduction  of  the  handwriting  trajectory  is  implemented  by  attaching  the 
sensor  to  a  pen  as  shown  in  Figure  5. 
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Figure  4.  Handwriting  Tracking  in  2-D. 


Figure  5.  Handwriting  Tracking  in  3-D. 


B.  THESIS  OBJECTIVES 

The  main  goal  of  this  thesis  is  to  construct  a  pen  input  device  using  an 
inertial/magnetic  sensor  module  for  the  reproduction  of  the  handwriting  trajectory  of 
alphanumeric  characters,  which  requires  the  following  to  be  achieved: 
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•  Conduct  an  experiment  in  2-D  plane  using  the  sensor  with  three  different 
motion  speeds, 

•  Implement  the  calibration  algorithm, 

•  Apply  the  velocity  correction  algorithm, 

•  Carry  out  experiments  in  2-D  plane  and  reproduction  of  the  handwriting 
trajectory, 

•  Perform  experiments  in  3-D  space  and  reproduction  of  the  handwriting 
trajectory. 

C.  THESIS  OUTLINE 

Chapter  II  describes  an  experiment  that  took  place  in  the  2-D  plane  and  explains 
that  the  writing  speed  must  be  relatively  high  for  an  accurate  handwriting  trajectory. 

Chapter  III  conducts  the  calibration  algorithm  for  computing  the  constant  length 
vector  pMi  between  the  nose  of  the  pen  and  the  point  where  the  inertial/magnetic  sensor 

module  is  attached  to  the  pen  and  eventually  for  the  computation  of  the  velocity  and 
finally  the  position  in  the  nose  of  the  pen. 

Chapter  IV  explains  the  velocity  correction  algorithm  for  eliminating  the  error  in 
acceleration  measurements  and  eventually  providing  accurate  handwriting  trajectory. 

Chapter  V  presents  experimental  results  of  the  handwriting  trajectory  in  a  2-D 

plane. 

Chapter  VI  presents  experimental  handwriting  results  for  tracking  alphanumeric 
characters  in  3-D  space. 

The  final  chapter  of  this  thesis  research  presents  conclusions  and 
recommendations  for  future  work. 
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II.  SENSOR  MOTION  VELOCITY 


This  chapter  describes  an  experiment  and  explains  with  results  that  for  the  MEMS 
sensor  used  in  the  experiment,  writing  speed  is  crucial. 

A.  MEASUREMENTS  OF  A  STRAIGHT  LINE 

In  this  experiment,  the  MicroStrain  3DM-GX1  sensor  was  placed  in  a  2-D  plane 
and  then  was  moved  along  a  straight  line  with  three  different  speeds  (high,  medium,  and 
slow).  The  actual  distance  of  the  straight  line  was  28  cm.  The  high  speed  of  the  sensor 
motion  was  computed  around  42.4  cm/sec,  the  medium  speed  15.3  cm/sec,  and  the  slow 
speed  10.5  cm/sec.  All  these  speeds  of  the  sensor  motion  were  calculated  approximately 
by  knowing  the  number  of  samples  that  was  collected  from  the  sensor,  the  actual  distance 
that  was  tracked  from  the  sensor,  and  the  sampling  frequency  of  the  sensor. 

The  purpose  of  this  experiment  was  to  verify  the  actual  distance  of  the  straight 
line  without  using  the  quaternion.  The  accelerometer  data  were  used  only  in  this 
experiment.  It  is  known  that  in  order  to  compute  the  actual  distance  of  the  straight  line, 
the  accelerometer  data  provided  from  the  sensor  in  the  sensor  coordinate  system  must  be 
converted  to  the  earth  coordinate  system.  In  that  case,  the  quaternion  is  required  to 
implement  that  conversion.  However,  in  this  experiment  in  order  to  avoid  the  use  of  the 
quaternion,  the  conversion  of  acceleration  data  from  the  sensor  coordinate  to  the  earth 
coordinate  was  shunned  by  keeping  the  sensor  in  a  fixed  orientation. 

A  low  pass  filter  was  used  to  the  acceleration  data  provided  from  the  sensor  in 
order  to  smooth  the  accelerations.  Then,  the  velocity  correction  method  was  applied  for 
the  interval  of  the  sensor  motion.  By  applying  this  method,  the  integration  was 
implemented  only  for  the  sensor  motion  and  not  prior  or  after  the  sensor  motion.  In  that 
way,  the  error  that  was  inserting  and  growing  in  velocity  correspondence  with  time  was 
eliminated. 

The  acceleration  data  provided  from  the  sensor  and  the  computed  distance  of  the 
straight  line  with  high  velocity  of  the  sensor  motion  are  shown  in  Figure  6. 
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The  acceleration  for  high  velocity  of  the  sensor 
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Figure  6.  Results  for  High  Speed  of  the  Sensor  Motion. 

According  to  Figure  6,  the  computed  distance  of  the  straight  line  was  25.5  cm. 
The  computed  distance  also  had  a  declination  of  1.5  cm  in  the  horizontal  axis  compared 
with  the  actual  vertical  straight  line.  It  is  obvious  from  the  acceleration  data  that  the 
signal  was  very  intense.  The  signal-to-noise  ratio  (SNR)  was  computed  for  that  case  to  be 
23.4484  dB. 

In  Figure  7  the  acceleration  data  provided  from  the  sensor  and  the  computed 
distance  of  the  straight  line  with  medium  velocity  of  the  sensor  motion  are  shown. 
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The  acceleration  for  medium  velocity  of  the  sensor 
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Figure  7.  Results  for  Medium  Speed  of  the  Sensor  Motion. 

Regarding  this  figure,  the  distance  was  computed  to  be  39  cm,  which  was  not  so 
accurate.  Apart  from  that,  the  declination  in  the  horizontal  axis  was  4.4  cm,  which  was 
bigger  than  the  previous  case.  According  to  the  acceleration  data,  the  signal  was  less 
intense  than  the  previous  case.  The  signal-to-noise  ratio  (SNR)  was  computed  for  this 
case  to  be  9.7138  dB. 

Figure  8  presents  the  acceleration  data  provided  from  the  sensor  and  the  computed 
distance  of  the  straight  line  using  the  slow  speed  for  the  sensor  motion. 
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Figure  8.  Results  for  the  Slow  Speed  of  the  Sensor  Motion. 

In  this  figure,  it  is  shown  that  the  computed  distance  was  53  cm,  which  is 
unacceptable  when  compared  with  the  actual.  The  declination  as  shown  in  Figure  8  was 
5.62  cm,  which  was  bigger  than  the  one  calculated  from  the  first  case  with  the  sensor 
motion  at  high  speed.  From  the  acceleration  data,  it  is  noticed  that  the  signal  became 
weak  and  the  noise  was  relatively  significant.  The  signal-to-noise  ratio  (SNR)  was 
computed  in  this  case  to  be  3.353  ldB. 

In  regards  to  Figures  6,  7,  and  8,  it  is  obvious  that  the  computed  distance  was  not 
only  much  more  accurate  but  also  had  smaller  declination  in  the  horizontal  axis  for  high 
velocity  of  the  sensor  motion.  Apart  from  that,  from  the  acceleration  data  it  was  noticed 
that  the  signal  was  much  more  intense  and  the  noise  was  relatively  small  for  high  velocity 
of  the  sensor  motion.  This  is  also  verified  from  the  computed  values  of  signal-to-noise 
ratio  (SNR). 
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The  following  table  summarizes  the  results  from  the  sensor  motion  regarding  the 
three  different  velocities. 


Sensor 

velocities 

(cm/sec) 

Average  value  of 
acceleration 

(m/sec2) 

Computed 

Distance 

(cm) 

Declination 

(cm) 

Distance  errors 

j  (%) 

42.4 

0.8282g 

25.5 

1.5 

!  8.9 

15.3 

0.222  lg 

39 

4.4 

|  39.28 

10.5 

0.1683g 

53 

5.62 

89.28 

Table  1 .  Sensor  Error  with  Three  Different  Velocities. 

The  fact  that  the  distance  errors  and  the  declination  distances  grow  while  the 
velocity  of  the  sensor  motion  and  the  average  value  of  accelerations  become  smaller 
confirms  that  the  accuracy  in  computed  distances  is  accomplished  only  by  using  the 
MicroStrain  3DM-GX1  sensor  with  high  motion  velocity. 

B.  SUMMARY 

The  results  of  this  experiment  suggest  that  for  accurate  tracking  at  writing 
trajectory  with  the  MicroStrain  sensor,  writing  speed  must  be  relatively  high,  or  a  more 
accurate  accelerometer  sensor  must  be  used. 

The  next  chapter  describes  the  calibration  algorithm  for  the  computation  of  the 
constant  length  pm  between  the  attached  point  of  the  sensor  (A)  and  the  nose  of  the  pen 
input  device  point  (B). 
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III.  CALIBRATION  ALGORITHM 


The  calibration  algorithm  is  needed  as  the  first  part  of  the  algorithm  in  the 
reproduction  of  the  handwriting  trajectory  for  the  computation  of  the  velocity  in  the  nose 
of  the  pen  and  eventually  for  the  estimation  of  the  position  in  the  nose  of  the  pen  in  the 
earth  coordinate  system.  This  algorithm  provides  the  constant  length  pm  in  the  sensor 

coordinate  system  between  the  attached  point  of  the  sensor  (A)  and  the  nose  of  the  pen- 
input  device  point  (B),  as  shown  in  Figure  9. 


Figure  9.  The  Pen-input  Device  and  the  Constant  Length p ^  . 

The  calibration  process  is  as  follows.  The  user  holds  the  pen  motionless  against  a 
surface  and  arbitrarily  rotates  it  for  a  few  seconds  while  keeping  the  pen  tip  (point  B) 
fixed.  While  the  user  arbitrarily  rotates  the  pen,  the  sensor  of  the  pen  acquires 
acceleration,  magnetic  field,  and  angular  rate  data. 

The  calibration  algorithm  provides  the  velocity  in  the  nose  of  the  pen  vB  by  using 
the  following  equation: 

vB=vA  +  coxpm  (1) 
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where  vB  is  the  velocity  in  the  nose  of  the  pen  in  each  axis  x,y,z  in  the  sensor  coordinate 
system,  vA  is  the  velocity  of  the  sensor  in  each  axis  x,y,z  in  the  sensor  coordinate  system, 
co  is  the  angular  velocity  in  each  axis  x,y,z  and  p,M  is  the  constant  length  between  points 
A  and  B. 

The  angular  velocity  co  for  each  axis  x,y,z  is  provided  from  the  sensor  and  the 
velocity  vA  of  the  sensor  for  each  axis  x,y,z  is  obtained  by  integrating  the  acceleration 
measurement  data,  as  shown  below  in  Equation  2. 

vA  =  |  a  dt  (2) 

While  performing  the  calibration,  the  velocity  in  the  nose  of  the  pen  vB  is  zero. 
Hence,  in  Equation  (1),  the  unknown  variable  is  the  constant  length  pAB  .  In  Equation  (1), 
the  cross  product  ( coxp AB)  is  replaced  by  the  product  SwpAB,  where  Sm  is  the  skew 
matrix  associated  with  angular  velocity  co  for  each  axis  x,y,z  given  by: 

r  0  -coz  co  ^ 

S(0  =  coz  0  -cox  (3) 

v-®v  ^  0  y 

The  constant  length  pm  can  be  computed  from  the  following  equation: 

°  =  W+^Pab  (4) 

However,  the  rank  of  the  skew  matrix  S  is  2  and  it  is  not  invertible.  Therefore,  the 

estimation  of  the  constant  length  pm  can  not  be  computed  from  Equation  (4)  and  the 
least-square  method  is  used. 

Let  SVA  be  a  vector  composed  of  the  point  A  velocities  in  the  sensor  coordinate 
system  for  each  axis  along  with  the  time  and  S  is  the  skew  matrix  as  described  above 
with  the  angular  rates  to  each  axis  along  with  time  too. 
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where  n  is  the  number  of  data  taken  from  the  sensor. 

Hence,  the  solution  of  the  constant  length  pm  is  given  by: 

pAB  =  -(sTsy1sTSvA  (7) 

The  implementation  of  the  calibration  algorithm  in  Matlab  code  is  described  in 
the  following  eight  steps. 

1.  Load  the  acceleration,  magnetic  field,  and  angular  rate  data  collected  by 
the  sensor  during  the  calibration  procedure. 

2.  Implement  a  low  pass  filter  for  the  acceleration  data  for  each  axis  x,y,z 
provided  from  the  sensor  in  order  to  separate  the  gravity  from  the 
acceleration  by  using  the  following  equation.  This  is  necessary  since  the 
accelerometers  of  the  sensor  sense  the  gravity  and  the  acceleration  due  to 
the  motion. 

“filtered  =  «  *  “measured  +(!“«)*  « filtered  (8) 

where  a  ftUered  is  the  fdtered  accelerations,  ameasured  is  the  measured  accelerations 

provided  from  the  sensor  and  the  variable  a  takes  a  value  from  0  to  1.  In  this  thesis 
research,  a  proper  value  for  a  is  0. 1073. 

3.  Calculate  the  quaternion  q  from  the  factored  quaternion  algorithm  [4] 
using  the  acceleration  and  magnetometer  data. 

4.  Convert  the  filtered  acceleration  measurements  from  the  sensor  coordinate 
system  into  the  earth  coordinate  system  using  the  following  equation: 

ea  =  q  sa  q  (9) 
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where  q *  is  the  conjugate  quaternion  of  q  and  the  product  (q  sa  q*)  is  called  the 
quaternion  product.  This  conversion  is  necessary  as  described  in  Chapter  I  for  the 
estimation  of  the  velocity  and  position  in  the  nose  of  the  pen  in  the  earth  coordinate 
system. 

5.  Obtain  the  velocity  vA  in  the  earth  coordinate  system  for  the  sensor  in 

each  axis  x,y,z  by  integrating  the  converted  filtered  accelerations  from 
step  4. 

evA=jeadt  (10) 

6.  Convert  evA  to  svA  and  compute  the  vector  svA  as  described  in  Equation 
(5)  for  all  number  of  data  n  taken  from  the  sensor. 

7.  Compute  the  skew  matrix  S  as  described  in  Equation  (6)  for  all  number  of 
data  n . 

8.  Provide  the  constant  length  pM>>  by  implementing  Equation  (7). 

All  the  above  steps  in  the  Matlab  code  are  implemented  by  using  a  function 
constant_vector=calibration(a,m,w),  where  a  is  an  n  x  4  matrix  with  time  and 
acceleration  measurements,  m  is  an  «x4  matrix  with  time  and  magnetic  field 
measurements,  and  co  is  an  n  x  4  matrix  with  time  and  angular  rate  measurements  taken 
during  the  calibration  procedure. 

In  this  thesis  research,  the  MicroStrain  3DM-GX1  sensor  was  attached  to  a  pen  in 
a  distance  of  14  centimeters  from  the  nose  of  the  pen.  After  the  implementation  of  the 
calibration  algorithm  through  the  steps  described  above,  in  Matlab  the  constant  length 
pm  was  computed  to  be  14.1  cm.  The  result  is  acceptable  and  very  close  to  the  actual 
distance  which  is  14  centimeters. 

Consequently,  now  knowing  the  constant  length  pm  between  point  A  and  point 
B,  the  velocity  vB  in  the  nose  of  the  pen  is  obtained  from  Equation  (1).  Finally,  the 
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estimation  for  the  position  pB  of  the  nose  of  the  pen  and  eventually  the  handwriting 
trajectory  is  obtained  by  integrating  the  velocity  vB  . 

ePs=\evBdt  (U) 

This  chapter  explained  the  calibration  algorithm  for  computing  the  constant  length 
pm  between  the  nose  of  the  pen  and  the  point  where  the  inertial/magnetic  sensor  module 
is  attached  to  the  pen  and  eventually  for  the  computation  of  the  velocity  and  the  position 
in  the  nose  of  the  pen.  The  following  chapter  describes  how  the  velocity  vB  in  the  nose  of 

the  pen,  before  the  integration  and  computation  of  position  pB ,  needs  correction  for  the 
reasons  that  were  explained  in  Chapter  I. 
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IV.  VELOCITY  CORRECTION 


This  chapter  describes  the  necessity  of  the  velocity  correction  in  the  main 
algorithm  for  the  representation  of  the  handwriting  trajectory.  An  experiment  is  described 
in  this  chapter  and  plots  are  provided  which  prove  that  the  velocity  correction  is 
necessary  to  obtain  the  accurate  handwriting  trajectory. 

In  this  experiment,  the  MicroStrain  3DM-GX1  sensor  was  placed  in  a  2-D  plane 
and  then  moved  and  tracked  along  a  straight  line  of  28  cm. 

In  Figure  10,  the  acceleration  plot  and  the  velocity  plot  of  the  sensor  in  one  axis  in 
the  sensor  coordinate  system  are  shown.  The  velocity  of  the  sensor  is  provided  from  the 
integration  of  the  acceleration  data. 
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Figure  10.  The  Acceleration  and  Velocity  Data  of  the  Sensor. 
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It  is  obvious  from  the  plot  of  the  acceleration  data  that  the  beginning  of  the  sensor 
motion  occurred  around  the  point  at  80  samples  and  the  end  of  the  sensor  motion 
happened  around  the  point  at  120  samples.  However,  it  is  clear  from  the  velocity  plot  that 
the  sensor’s  velocity  is  nonzero  prior  to  the  point  at  80  samples  and  after  the  point  at  120 
samples.  This  is  an  error,  because  prior  to  the  sensor  motion  that  started  around  the  point 
at  80  samples  and  at  the  end  of  the  sensor  motion  that  happened  around  the  point  at  120 
samples,  the  sensor  velocity  should  be  zero.  The  integration  of  this  nonzero  data  in  order 
to  obtain  the  handwriting  trajectory  of  the  straight  line  results  in  an  error  in  the 
reproduction  of  the  handwriting  trajectory. 

Apart  from  that,  it  was  explained  in  Chapter  II  that  the  noise  of  the  sensor  affects 
the  accuracy  of  the  handwriting  trajectory  and  consequently  the  position  of  the  pen. 
Consequently,  this  error  occurs  not  only  from  the  integration  of  the  nonzero  data  but  also 
from  the  noise  of  the  sensor.  It  is  called  drift  and  it  is  growing  along  with  the  time  as  it  is 
observed  in  the  velocity  plot  in  Figure  10. 

Figure  11  shows  that  the  handwriting  trajectory  of  the  straight  line  not  only  is 
inaccurate  but  also  has  an  error  at  the  beginning  and  at  the  end  of  the  handwriting 
trajectory.  Consequently,  the  implementation  of  the  velocity  correction  is  needed  in  the 
main  algorithm  for  the  elimination  of  that  error  and  for  the  reproduction  of  an  accurate 
handwriting  trajectory. 
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The  vertical  straight  line  without  the  implementation  of  the  velocity  correction 
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Figure  1 1 .  The  Writing  Trajectory  of  the  Vertical  Straight  Line. 

In  the  Matlab  code,  the  process  for  the  implementation  of  the  velocity  correction 
is  the  following.  First,  a  threshold  value  is  assigned  and  whenever  the  measured 
acceleration  is  below  the  threshold  value  the  sensor  is  considered  motionless;  otherwise 
the  sensor  has  motion  and  acquires  data.  Flence,  in  the  implementation  of  the  threshold 
value,  the  starting  and  ending  sensor  motion  points  are  decided  and  the  nonzero  values  in 
the  sensor  velocity  are  eliminated. 

Next,  the  elimination  of  the  sensor  noise  for  the  interval  between  the  starting  and 
ending  sensor  motion  points  is  accomplished  by  implementing  Equation  (15).  The  proof 
of  Equation  (15)  is  presented  below. 

The  measured  acceleration  is  given  by  the  following  equation: 

am{t)  =  at(t)  +  e  (12) 
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where  am(t)  is  the  measured  acceleration  provided  from  the  sensor,  a,(t)  is  the  actual 
acceleration,  and  e  is  the  error  coming  from  the  sensor  noise. 

The  velocity  is  obtained  by  integrating  Equation  (12)  as  follows: 

v,n  (0  =  {  a,n  (t)dt  =  ^(at(t)  +  e)dt 
=  va(t)  +  et 

where  va(t)  is  the  actual  velocity,  vm(t)  is  the  computed  velocity  and  the  term  (et  )  means 
that  the  error,  grows  along  with  time  and  t  is  the  time  interval. 

The  end  of  the  sensor  motion  occurs  at  time  tx  where  the  velocity  vjt)  is  zero. 
Hence,  Equation  (13)  now  gives  the  value  of  that  error  as  shown  below. 

vm(ti)=0  +  eti  =>  e  =^tl  (14) 

h 

Now,  substituting  Equation  (14)  into  Equation  (13)  provides  the  corrected 
velocity  va(t)  for  the  interval  between  the  starting  and  ending  sensor  motion  points. 

v,(0=v,(0-N^»  (15) 

h 

Hence,  by  implementing  Equation  (15)  in  Matlab  code,  the  velocity  correction 
method  is  applied  and  the  evidence  that  the  effect  of  drift  is  eliminated  is  shown  in  the 
following  figure  (Figure  12). 
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The  acceleration  of  the  sensor  in  one  axis 
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The  corrected  velocity  of  the  sensor  in  one  axis 


Figure  12.  The  Corrected  Velocity  of  the  Sensor. 

It  is  observed  from  Figure  12  that  prior  to  the  sensor  motion  and  after  the  sensor  motion 
its  velocity  is  zero. 

The  accuracy  of  the  handwriting  trajectory  and  consequently  the  need  for  the 
velocity  correction  is  presented  in  Figure  13,  where  the  distance  of  the  straight  line  is 
25.5  cm  and  is  acceptable  as  the  actual  distance  is  28  cm. 
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The  vertical  straight  line  with  the  velocity  correction  method  applied 
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Figure  13.  The  Vertical  Straight  Line  Constructed  with  the  Velocity  Correction 

Algorithm. 


Hence,  it  is  obvious  that  the  implementation  of  the  velocity  correction  method  is 
necessary  in  the  main  algorithm  of  this  thesis  research  for  the  accurate  reproduction  of 
the  handwriting  trajectory. 

This  chapter  described  the  velocity  correction  algorithm  in  order  to  eliminate  the 
drift  in  the  acceleration  measurements  and  to  provide  accurate  handwriting  trajectory. 
The  next  two  chapters  demonstrate  the  handwriting  trajectories  of  alphanumeric 
characters  for  handwriting  tracking  in  2-D  space  and  3-D  plane. 
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V.  HANDWRITING  TRAJECTORY  IN  A  2-D  PLANE 


This  chapter  presents  the  experimental  results  of  the  handwriting  trajectory  in  a  2- 
D  plane  for  alphanumeric  characters  in  order  to  show  the  effectiveness  of  the  proposed 
algorithm. 

The  MicroStrain  3DM-GX1  sensor  was  placed  in  a  2-D  plane  and  started  to  move 
and  track  the  alphanumeric  characters  with  high  motion  velocity  in  order  to  acquire 
accurate  handwriting  trajectory  as  explained  in  Chapter  II.  Before  the  presentation  of  the 
reproduced  handwriting  results  in  a  2-D  plane,  it  is  necessary  to  explain  the  main 
algorithm  that  was  implemented  in  Matlab  in  the  following  seven  steps. 

1.  Loaded  the  acceleration  and  quaternion  data  provided  by  the  sensor  during  the 
sensor  motion  for  tracking  the  alphanumeric  characters.  It  is  necessary  to 
mention  that  for  these  experiments  the  sensor  quaternion  was  used  instead  of  the 
quaternion  provided  from  the  factored  quaternion  algorithm  (FQA). 

2.  Implemented  a  low-pass  filter  for  the  acceleration  data  for  each  axis  x,y  provided 
from  the  sensor  in  order  to  separate  the  gravity  from  the  acceleration  as  explained 
in  the  calibration  algorithm  in  Chapter  III  by  using  Equation  (8).  For  the 
reproduction  of  the  handwriting  trajectory  the  proper  value  for  a  is  0. 1073. 

3.  Converted  the  filtered  accelerations  measurements  from  the  sensor  coordinate 
system  into  the  earth  coordinate  system  using  Equation  (9)  from  Chapter  III. 

4.  Obtained  the  velocity  vsemor  in  the  earth  coordinate  system  for  the  sensor  in  each 
axis  x,y  by  integrating  the  converted  filtered  accelerations. 

ev  =  f  eadt  (16) 

sensor  J  V  / 

5.  Applied  the  velocity  correction  method  as  explained  in  Chapter  IV  in  order  to 
obtain  an  accurate  handwriting  trajectory. 
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6.  Implemented  a  low  pass  filter  for  the  corrected  velocity  from  step  5  using  once 
again  Equation  (8)  from  Chapter  III  in  order  to  acquire  not  only  an  accurate  but 
also  a  smooth  handwriting  trajectory. 

7.  Obtained  the  position  of  the  sensor  and  represented  the  handwriting  trajectory  by 
integrating  the  velocity  evsensor  of  the  sensor. 

ep  =\ev  dt  (17) 

Jr  sensor  J  sensor  v  ’ 

Figures  14,  15,  16,  17,  18,  19,  and  20  show  the  reproduced  handwriting  trajectory 
of  the  letters  A,  B,  C,  M,  N,  P,  and  S. 
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Figure  14.  The  Handwriting  Trajectory  of  the  Letter  A,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  letter  B 
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The  Handwriting  Trajectory  of  the  Letter  B,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  Handwriting  Trajectory  of  the  Letter  C,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


29 


0.16 


The  letter  M 
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Figure  17.  The  Handwriting  Trajectory  of  the  Letter  M,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  18.  The  Handwriting  Trajectory  of  the  Letter  N,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  letter  P 
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Figure  19.  The  Flandwriting  Trajectory  of  the  Letter  P,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  20.  The  Handwriting  Trajectory  of  the  Letter  S,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figures  21,  22,  23,  24,  25,  26,  27,  28,  29,  and  30  present  the  reproduced 
handwriting  trajectory  of  the  numbers  1  to  10. 
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Figure  21.  The  Handwriting  Trajectory  of  the  Number  1,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  22.  The  Handwriting  Trajectory  of  the  Number  2,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  3 
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Figure  23.  The  Handwriting  Trajectory  of  the  Number  3,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  24.  The  Handwriting  Trajectory  of  the  Number  4,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  5 
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Figure  25.  The  Handwriting  Trajectory  of  the  Number  5,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  26.  The  Handwriting  Trajectory  of  the  Number  6,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  7 
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Figure  27.  The  Handwriting  Trajectory  of  the  Number  7,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  28.  The  Handwriting  Trajectory  of  the  Number  8,  with  the  Horizontal  and 

Vertical  Axis  Units  in  Meters. 
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The  number  9 
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Figure  29.  The  Handwriting  Trajectory  of  the  Number  9,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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Figure  30.  The  Handwriting  Trajectory  of  the  Number  10,  with  the  horizontal  and 

vertical  axis  units  in  meters. 
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It  is  known  that  there  are  some  alphanumeric  characters  such  as  the  letter  A  and 
the  number  7  that  can  be  written  in  more  than  one  stroke.  For  these  cases  in  the  main 
algorithm  the  start  and  end  points  of  each  stroke  are  found,  and  consequently  the  velocity 
correction  method  and  then  the  integration  of  the  corrected  velocity  are  implemented 
separately  for  each  stroke. 

In  this  thesis  research,  the  sensor  was  also  tested  to  reproduce  the  handwriting 
trajectory  of  a  word/acronym  in  a  2-D  plane.  The  following  figure  (Figure  31)  presents 
the  recognizable  handwriting  trajectory  of  the  word/acronym  NPS. 
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Figure  3 1 .  The  Handwriting  Trajectory  of  the  acronym  NPS,  with  the  horizontal  and 

vertical  axis  units  in  meters. 

Figure  31  is  evidence  that  the  sensor  can  reproduce  handwriting  trajectories  not 
only  for  letters  but  also  for  words  and  acronyms  as  well  as  sentences.  Hence,  it  is  obvious 
from  all  the  above  figures  that  the  reproduced  handwriting  trajectories  in  a  2-D  plane  are 
sufficient  and  recognizable  by  humans. 
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VI.  HANDWRITING  TRAJECTORY  IN  A  3-D  SPACE 


This  chapter  presents  the  experimental  handwriting  results  for  tracking 
alphanumeric  characters  in  3-D  space  by  using  the  pen  input  device. 

The  MicroStrain  3DM-GX1  sensor  was  attached  to  a  pen  at  a  distance  of  14  cm 
from  the  nose  of  the  pen,  and  the  constant  length  pAB  between  the  attached  point  of  the 

sensor  (A)  and  the  nose  of  the  pen  input  device  (B)  was  computed  to  be  14.1  cm  by 
implementing  the  calibration  algorithm  in  Matlab  as  explained  in  Chapter  III. 

The  main  algorithm  used  for  the  reproduction  of  the  handwriting  trajectory  by 
using  the  pen  input  device  is  described  in  the  following  eight  steps. 

1 .  Loaded  the  acceleration,  quaternion,  and  angular  rate  data  provided  by  the  sensor 
during  the  pen’s  motion  for  tracking  the  alphanumeric  characters.  For  these 
handwriting  results,  the  sensor  quaternion  was  also  used  instead  of  the  quaternion 
provided  from  the  factored  quaternion  algorithm  (FQA). 

2.  Implemented  a  low  pass  filter  for  the  acceleration  data  for  each  axis  x,y,z  provided 
from  the  sensor  in  order  to  separate  the  gravity  from  the  acceleration  as  explained 
in  the  calibration  algorithm  in  Chapter  III  by  using  Equation  (8).  For  the 
reproduction  of  handwriting  trajectories  the  proper  value  for  a  is  0. 1073. 

3.  Converted  the  filtered  accelerations  measurements  from  the  sensor  coordinate 
system  into  the  earth  coordinate  system  using  Equation  (9)  from  Chapter  III. 

4.  Obtained  the  velocity  vA  in  the  earth  coordinate  system  for  the  sensor  in  each  axis 
x,y,z  by  integrating  the  converted  filtered  accelerations. 

evA=feadt  (18) 

5.  Computed  the  velocity  vB  in  the  nose  of  the  pen  in  the  earth  coordinate  system 
based  on  the  sensor  velocity  vA  by  using  the  following  equation. 

eVB  =  eVA+qCa>X’PAB)?t  (19) 
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where  co  was  the  angular  rate  provided  from  the  sensor  and  pAB  was  the  constant  length 

computed  from  the  calibration  algorithm  as  explained  in  Chapter  III.  The  quaternion 
rotation  operator  was  used  in  order  to  convert  the  angular  rate  vector  co  and  the  constant 
length  vector  pm  from  the  sensor  coordinate  system  into  the  earth  coordinate  system. 

6.  Applied  the  velocity  correction  algorithm  in  order  to  recognize  the  pause  phases  in 
writing  and  obtain  an  accurate  handwriting  trajectory  as  explained  in  Chapter  IV. 

7.  Implemented  a  low  pass  filter  for  the  corrected  velocity  from  step  6  using 
Equation  (8)  from  Chapter  III  in  order  to  acquire  an  accurate  and  smooth 
handwriting  trajectory. 

8.  Obtained  the  position  in  the  nose  of  the  pen  in  the  earth  coordinate  system  and 
reproduced  the  handwriting  trajectory  by  integrating  the  corrected  velocity  vB  in 
the  nose  of  the  pen. 

epB=\evBdt  (20) 

Figures  32,  33,  34,  35,  36,  37,  38,  and  39  present  the  reproduced  handwriting 
trajectories  of  the  letters  A,  B,  C,  E,  F,  N,  P,  and  S. 
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The  letter  A 


0.12 

0.1 

0.08 

0.06 

0.04 

0.02 

0 

-0.02 

-0.04 

Figure  32.  The  Handwriting  Trajectory  of  the  Letter  A,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


0.2 

0.15 

o.i 

0.05 


Figure  33.  The  Handwriting  Trajectory  of  the  Letter  B,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


The  letter  B 


-0.1  -0.05  0  0.05  0.1  0.15  0.2  0.25  0. 


i - 1 - ( - 1 - 1 - 1 - 1 - 1 - 1 - 1 - r 


i _ i _ i _ i _ i _ i _ i _ i _ i _ i _ i _ 

-0.06  -0.04  -0.02  0  0.02  0.04  0.06  0.08  0.1  0.12  0.14 
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0.04 


The  letter  C 


0.02  - 

0  - 

-0.02  - 

-0.04  - 

-0.06  - 

-0.08  - 

-0.1  - 

_0  12  I _ i _ i _ i _ i _ i _ i _ i _ 

-0.1  -0.08  -0.06  -0.04  -0.02  0  0.02  0.04  0.06 

Figure  34.  The  Handwriting  Trajectory  of  the  Letter  C,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


0.05 


o 


-0.05 


-o.i 


-0.15 


-0.2 

-0.1  -0.08  -0.06  -0.04  -0.02  0  0.02  0.04 


The  letter  E 

i  (  (  i  i  i  r 


j _ i _ i _ i _ i _ i _ l 


Figure  35.  The  Handwriting  Trajectory  of  the  Letter  E,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  letter  F 


0.04 

0.02 

0 

-0.02 

-0.04 

-0.06 

-0.08 

-0.1 

-0.12 

-0.14 

-0.16 


Figure  36.  The  Flandwriting  Trajectory  of  the  Letter  F,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


1  I  I  I  I  I  I  I  I  I  r 


_ I _ I _ I _ [ _ I _ I _ [ _ I _ I _ [ _ [ _ 

-0.16  -0.14  -0.12  -0.1  -0.08  -0.06  -0.04  -0.02  0  0.02  0.04 


The  letter  N 


0.14 

0.12 

0.1 

0.08 

0.06 

0.04 

0.02 

0 

-0.02 

-0.04 


-0.05  0  0.05  0.1  0.15  0.2 


Figure  37.  The  Handwriting  Trajectory  of  the  Letter  N,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  letter  P 


0.16  - 
0.14  - 
0.12  - 
0.1  - 
0.08  - 
0.06  - 
0.04  - 
0.02  - 
0  - 

-0.02 - 1 - t - 1 - 1 - 1 - t - 1 - 

-0.04  -0.03  -0.02  -0.01  0  0.01  0.02  0.03  0.04 


Figure  38.  The  Handwriting  Trajectory  of  the  Letter  P,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


0.05 

o 

-0.05 

-o.i 

-0.15 

-0.2 

-0.25 

-0.3 


Figure  39.  The  Handwriting  Trajectory  of  the  Letter  S,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


The  letter  S 


-0.05  0  0.05 
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Figures  40,  41,  42,  43,  44,  45,  46,  47,  48,  and  49  are  the  reproduced  handwriting 
trajectories  of  the  numbers  1  to  10. 


The  number  1 

I  I  I  I  I  I  I  I 

0.08  - 
0.06  - 
0.04  - 
0.02  - 
0  - 

-0.02  - 
-0.04  - 
-0.06  - 
-0.08  - 
-0.1  - 
-0.12  - 

_ i _ i _ i _ i _ i _ i _ i _ i _ 

-0.06  -0.04  -0.02  0  0.02  0.04  0.06  0.08  0.1  0.12 


Figure  40.  The  Handwriting  Trajectory  of  the  Number  1 ,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


The  number  2 


0.05 


O 


-0.05 


-0.1 


-0.15 


-0.2 


-0.06  -0.04  -0.02  O  0.02  0.04  0.06 


Figure  41 .  The  Handwriting  Trajectory  of  the  Number  2,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  3 


0.04 

0.02 

0 

-0.02 

-0.04 

-0.06 

-0.08 

-0.1 

-0.12 
-0. 

Figure  42.  The  Handwriting  Trajectory  of  the  Number  3,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


o 

-0.05 

-o.i 

-0.15 

-0.2 

-0.15  -0.1  -0.05  0  0.05  0.1  0.15 


The  number  4 

i  i  !  i  !  i  r 


j _ i _ i _ i _ i _ i _ l 


n: i i i [ i i 


_i _ 1 _ i _ 


_ 1 _ i _ 


_ 1 _ i _ 


06  -0.04  -0.02  0  0.02  0.04  0.06  0.08  0.1  0.12 


Figure  43.  The  Handwriting  Trajectory  of  the  Number  4,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  5 


0.05 


0 


-0.05 


-0.1 


-0.15 


Figure  44.  The  Handwriting  Trajectory  of  the  Number  5,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


0.05 


o 


-0.05 


-o.i 


-0.15 


-0.12  -0.1  -0.08  -0.06  -0.04  -0.02  0  0.02  0.04 


The  number  6 

1  i  i  i  i  i  i  r 


j _ i _ i _ i _ i _ i _ i _ l 


i - 1 - 1 - 1 - 1 - [ - r 


_ l _ 1 _ l _ l _ l _ l _ l _ 

-0.08  -0.06  -0.04  -0.02  0  0.02  0.04 


Figure  45.  The  Handwriting  Trajectory  of  the  Number  6,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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The  number  7 


0.04 

0.02 

0 

-0.02 

-0.04 

-0.06 

-0.08 

-0.1 

-0.12 

-0.14 

-0.16 


-0.06  -0.04  -0.02  0  0.02  0.04  0.06  0.08  0.1  0.12 


Figure  46.  The  Handwriting  Trajectory  of  the  Number  7,  with  the  horizontal  and  vertical 

axis  units  in  meters. 


The  number  8 


0.04 

0.02 

0 

-0.02 

-0.04 

-0.06 

-0.08 

-0.1 

-0.12 

-0.14 

-0.16 


-0.15 


-0.1 


-0.05 


0.05 


0.1 


Figure  47.  The  Handwriting  Trajectory  of  the  Number  8,  with  the  horizontal  and  vertical 

axis  units  in  meters. 
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0.08 


The  number  9 


0.06  - 
0.04  - 
0.02  - 
0  - 

-0.02  - 
-0.04  - 
-0.06  - 

-0.08  -  J? 

-0.1  - 

_0  12 _ i _ i _ i _ i _ i _ i 

-0.06  -0.04  -0.02  0  0.02  0.04  0.06 


Figure  48.  The  Handwriting  Trajectory  of  the  Number  9,  with  the  horizontal  and  vertical 

axis  units  in  meters. 

The  number  10 

0.08 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 

0.06  - 

0.04  - 

/  )  /  \ 

0.02  - 

,  '  / 

0  - 

I  / 

-0.02  - 

/  \  / 

-0.04  - 

-0.06  - 

-0.08  I - 1 - i - 1 - 1 - 1 - i - 1 - 

-0.22  -0.2  -0.18  -0.16  -0.14  -0.12  -0.1  -0.08  -0.06 


The  number  10 

T  I  I  I  I  I  r 


/ 

/ 


/ 


I _ I _ [ _ I _ I _ I _ [ _ L 


Figure  49.  The  Handwriting  Trajectory  of  the  Number  10,  with  the  horizontal  and 

vertical  axis  units  in  meters. 
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For  the  reasons  explained  in  Chapter  V,  in  this  main  algorithm  the  beginning  and 
the  end  points  of  each  stroke,  are  also  found  and  the  velocity  correction  algorithm  and 
then  the  integration  of  the  corrected  velocity  are  implemented  separately  for  each  stroke. 

Figure  50  presents  the  handwriting  trajectory  of  the  acronym  NPS  by  tracking  this 
word  with  the  pen  input  device. 

The  word  NPS 

0.2  i  i  i  i  i  i  i  i  i  i  i 

0.15- 

0.1  - 

0.05  - 

0  - 

_0  05  ■  1 _ I _ £ _ I _ 1 _ I _ 1 _ 1 _ £ _ I _ Jg  V  i. 

-0.25  -0.2  -0.15  -0.1  -0.05  0  0.05  0.1  0.15  0.2  0.25 


Figure  50.  The  Ffandwriting  Trajectory  of  the  Acronym  NPS,  with  the  horizontal  and 

vertical  axis  units  in  meters. 

Figure  50  indicates  that  the  pen  input  device  can  be  used  for  the  reproduction  of 
handwriting  trajectories  not  only  for  letters  but  also  for  words  as  well  as  sentences. 

Chapters  V  and  VI  presented  experimental  handwriting  results  of  alphanumeric 
characters  in  2-D  space  and  3-D  plane.  All  the  matlab  code  that  was  used  in  this  thesis 
research  is  available  to  Professor  Yun  Xiaoping.  The  next  chapter  summarizes  the 
research  conducted  in  this  thesis,  and  provides  recommendations  for  future  work. 
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VII.  CONCLUSION  AND  RECOMMENDATIONS 


A.  CONCLUSION 

In  this  thesis  research,  the  feasibility  of  a  pen  input  device  that  can  reproduce  the 
handwriting  trajectory  of  alphanumeric  characters  using  an  inertial/magnetic  sensor 
module  was  investigated. 

This  research  was  started  with  an  experiment  demonstrating  the  importance  of  the 
sensor  velocity  for  the  reproduction  of  an  accurate  handwriting  trajectory.  The  sensor  was 
placed  in  a  2-D  plane  and  moved  to  track  a  straight  line  with  three  different  velocities. 
The  numerical  and  graphical  results  indicated  that  the  high  sensor  velocity  is  crucial  for 
reproducing  handwriting  trajectory. 

In  this  thesis,  a  calibration  algorithm  was  also  studied.  The  calibration  algorithm 
was  computed  the  constant  length  vector  between  the  nose  of  the  pen  input  device  and 
the  point  where  the  inertial/magnetic  sensor  was  attached  to  the  pen.  In  this  research,  the 
sensor  was  attached  at  a  pen  in  a  distance  of  14  cm  from  the  nose  of  the  pen.  After  the 
implementation  of  the  equations  that  described  in  Matlab,  the  constant  length  was 
computed  to  be  14.1  cm. 

A  velocity  correction  algorithm  was  also  described  in  this  thesis.  An  experiment 
with  the  sensor  indicated  that  prior  to  the  sensor  motion  and  after  the  sensor  motion,  the 
sensor  velocity  was  nonzero.  The  nonzero  data  in  the  velocity  result  in  an  error  or  drift 
and  grows  along  with  the  time.  Mathematical  equations  were  applied  for  eliminating  that 
error  and  after  the  implementation  of  them  in  Matlab,  an  accurate  handwriting  trajectory 
was  accomplished. 

Experimental  handwriting  results  of  alphanumeric  characters  for  handwriting 
tracking  in  2-D  plane  and  3-D  space  were  presented  in  this  thesis  research.  It  is  observed 
from  all  the  handwriting  results  that  the  reproduced  handwriting  trajectories  are 
qualitatively  correct  and  recognizable  by  humans.  This  indicates  that  the  construction  of 
this  pen  input  device  using  an  inertial/magnetic  sensor  for  the  reproduction  of  the 
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handwriting  trajectory  of  alphanumeric  characters  to  a  2-D  plane  can  be  feasible. 
However,  the  inaccurate  handwriting  trajectories  result  from  the  inaccurate  quaternion 
provided  from  the  sensor  and  the  speed  of  the  writing  motion. 

As  explained  in  Chapter  II,  the  sensor  provides  accurate  acceleration  data  and 
consequently  an  accurate  handwriting  trajectory  only  by  using  the  sensor  with  high 
motion  velocity.  As  mentioned  in  Table  1  of  Chapter  II,  the  handwriting  trajectory  of  a 
straight  line  has  the  smaller  distance  error  for  the  sensor  motion  velocity  around  42.4 
cm/sec.  However,  the  performance  of  tracking  alphanumeric  characters  in  a  2-D  plane 
and  3-D  space  with  a  sensor  motion  velocity  around  40cm/sec  to  50cm/sec  is  really 
difficult  and  sometimes  infeasible,  especially  for  complicated  alphanumeric  characters. 
Apart  from  that,  the  handwriting  tracking  is  not  accurate  with  such  a  writing  speed,  and 
consequently  the  sensor  does  not  collect  accurate  data  for  the  alphanumeric  characters. 
Hence,  the  sensor  does  not  provide  the  accurate  acceleration  data  which  leads  to  errors  in 
the  double  integration  and  consequently  an  inaccurate  handwriting  trajectory  is 
reproduced. 

Many  times  it  was  explained  and  implemented  in  this  thesis  research  that  the 
quaternion  is  necessary  for  the  conversion  of  the  acceleration  measurements  from  the 
sensor  coordinate  system  into  the  earth  coordinate  system  by  using  Equation  (9)  from 
chapter  III.  This  conversion  is  necessary  because  the  position  of  the  nose  of  the  pen  must 
be  calculated  in  the  earth  coordinate  system  as  explained  in  Chapter  I.  It  is  obvious  that 
errors  in  the  quaternion  will  affect  the  accuracy  of  the  acceleration  data  in  the  earth 
coordinate  system. 

The  sensor  specifications  as  described  in  [3],  state  that  the  dynamic  accuracy  of 
the  sensor  is  +/-  2  degrees  for  arbitrary  orientation  angles.  That  indicates  that  the  sensor 
does  not  provide  the  accurate  orientation  quaternion  and  thus  of  the  pen  while  the 
handwriting  tracking  is  taking  place.  Therefore,  the  inaccurate  quaternion  creates  errors 
in  the  acceleration  data  and  accordingly  after  the  double  integration  to  the  handwriting 
trajectory. 
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Apart  from  that,  the  performance  of  handwriting  tracking  of  alphanumeric 
characters  using  the  factored  quaternion  algorithm  (FQA)  was  also  tested  instead  of  the 
quaternion  provided  from  the  sensor.  However,  the  handwriting  results  were 
unacceptable  and  unrecognizable. 

This  was  expected  because  the  computation  of  the  quaternion  from  the  factored 
quaternion  algorithm  as  described  in  [4]  is  based  only  on  the  measured  acceleration  and 
magnetometer  data  as.  The  FQA  [4]  implements  equations  for  the  computation  of  the 
quaternion  using  the  measured  acceleration  and  magnetometer  data.  However,  it  is 
known  from  Chapter  II  that  the  sensor  did  not  provide  accurate  acceleration  data  for  a 
nonnal  hand’s  motion  velocities. 

Hence,  it  is  obvious  that  the  FQA  [4]  provided  the  quaternion  with  a  bigger  error 
than  the  sensor  quaternion  for  its  motion  because  it  depends  only  on  the  inaccurate 
acceleration  measurements.  For  that  reason,  the  FQA  was  used  only  in  the  calibration 
algorithm  as  stated  in  Chapter  III  because  handwriting  tracking  and  consequently  the 
sensor  motion  was  not  taking  place. 

B.  RECOMMENDATIONS 

For  the  conclusions  explained  above,  it  is  recommended  that  the  accuracy  of  the 
handwriting  trajectory  can  be  improved  by  using  a  different  kind  of  inertia/magnetic 
sensor.  It  is  suggested  to  use  an  inertia/magnetic  sensor  that  can  provide  accurate 
acceleration  data  to  slower  and  normal  hand  motion  velocities. 

It  is  also  recommended  for  the  improvement  of  the  accuracy  of  the  quaternion  an 
algorithm  that  can  be  more  effective  than  the  factored  quaternion  algorithm  (FQA)  be 
developed. 

Finally,  it  is  recommended  the  use  and  test  of  the  pen  input  device  be  conducted 
by  multiple  users. 
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APPENDIX 


The  following  specifications  for  the  inertial/magnetic  sensor  module  MicroStrain 
3DM-GX1  that  was  used  in  this  thesis  research  are  from  reference  [3]. 


Orientation  Range 

360°  full  scale  (FS),  all  axes  (Matrix,  Quaternion  modes) 

Sensor  Range 

Gyros:  300°/sec  FS;  accelerometers:  5  G's  FS;  magnetometers:  1.2  Gauss  FS 

A/D  Resolution 

16  bits 

Accelerometer 

Nonlinearity 

0.2% 

Accelerometer  Bias 
Stability* 

0.010  G's 

Gyro  Nonlinearity 

0.2% 

Gyro  Bias  Stability* 

0.7°/sec 

Magnetometer 

Nonlinearity 

0.4% 

Magnetometer  Bias 
Stability* 

.010  Gauss 

Orientation  Resolution 

<  0.1°  minimum 

Repeatability 

0.20° 

Accuracy 

0.5°  typical  for  static  test  conditions,  2°  typical  for  dynamic  (cyclic)  test 
conditions  and  for  arbitrary  orientation  angles 

Output  Modes 

Matrix,  Quaternion,  Euler  angles  and  9  scaled  sensors  with  temperature 

Digital  Outputs 

Serial  RS-232  and  RS-485  optional  with  software  programming 

Analog  Output  Option 

0-5  volts  FS  for  Euler  angles  (pitch  90,  roll  180,  yaw  360°) 

Digital  Output  Rates 

100  Hz  for  Euler,  Matrix,  Quaternion;  350  Hz  for  nine  orthogonal  sensors  only 

Serial  Data  Rate 

19.2/38.4/115.2  Kbaud,  software  programmable 

Supply  Voltage 

5.2  VDC  min.,  12  VDC  max. 

Supply  Current 

65  milliamps 

Connectors 

One  keyed  LEMO,  two  for  RS-485  option 

Operating  Temperature 

-40°  to  +70°  C  with  enclosure;  -40°  to  +85°  C  without  enclosure 

Size  with  Enclosure 

Size  without  Enel. 

65  x  90  x  25  mm  2.5  x  3.5  x  1.0" 

42  x  40  x  15  mm;  1.65  x  1.6  x  0.6" 

Weight 

74.6  gr.  with  enclosure,  25.8  gr.  without  enclosure 

Shock  Limit 

1000  G's  (unpowered);  500  G's  (powered) 

*  Accuracy  and  stability  specifications  obtained  over  operating  temperatures  of  -40°  to  +70°  C 
with  known  sine  and  step  inputs,  including  angular  rates  of  300  degrees  per  second. 
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